An Emergent Micro-Services Approach to Digital Curation Infrastructure

In order better to meet the needs of its diverse University of California (UC) constituencies, the California Digital Library UC Curation Center is re-envisioning its approach to digital curation infrastructure by devolving function into a set of granular, independent, but interoperable microservices. Since each of these services is small and self-contained, they are more easily developed, deployed, maintained, and enhanced; at the same time, complex curation function can emerge from the strategic combination of atomistic services. The emergent approach emphasizes the persistence of content rather than the systems in which that content is managemed, thus the paradigmatic archival culture is not unduly coupled to any particular technological context. This results in a curation environment that is comprehensive in scope, yet flexible with regard to local policies and practices and sustainable despite the inevitability of disruptive change in technology and user expectation. 1 This article is based on the paper given by the authors at iPRES 2009; received November 2009, published June 2010. The International Journal of Digital Curation is an international journal committed to scholarly excellence and dedicated to the advancement of digital curation across a wide range of sectors. ISSN: 1746-8256 The IJDC is published by UKOLN at the University of Bath and is a publication of the Digital Curation Centre. An Emergent Micro-Services Approach to Digital Curation Infrastructure 173 Introduction Information technology and resources have become both integral and indispensable to the pedagogic mission of the University of California (UC). Members of the UC community routinely produce and utilize a wide variety of digital assets in the course of teaching, learning, and research. These assets represent the intellectual capital of the University; they have inherent enduring value and need to be managed carefully to ensure that they will remain available for use by future scholars. Within the UC system the California Digital Library (CDL) UC Curation Center (UC3) has a broad mandate to ensure the long-term usability of the University’s digital assets. UC3 increasingly sees its mission in terms of digital curation, the set of policies and practices focused on maintaining and adding value to a body of trusted digital content for use now and into the indefinite future (Abbott, 2008). Traditionally, preservation and access have been considered disparate activities. Properly, however, they should be seen as complementary functions: preservation focused on ensuring use over time, while use depends upon preservation up to a point in time (Rusbridge, 2008). Curation is thus an ongoing process of management and enrichment at all stages of the lifecycle of a digital asset (Higgins, 2008). While curation is not solely a technical undertaking – curation success is, for example, highly dependent on important human competencies, analysis, and decision making – a robust infrastructure in which to manage valuable digital content efficiently and effectively is nevertheless a necessary foundation. Curation Infrastructure As a central system-wide service provider to the 10 UC campuses, UC3 is continually asked to assume stewardship responsibility for digital content in ever increasing number, size, and diversity of type. Furthermore, this content is often used and repurposed in novel contexts. Thus, the programmatic imperative of UC3 is to provide a curation environment that is comprehensive in scope, yet flexible with regard to local policies and practices, the inevitability of disruptive change in technology and user expectation, and the realization that curation over archival timespans is a relay (Janée, Frew, & Moore, 2008). To achieve this goal, UC3 believes it is necessary to deprecate the centrality of the curation repository as place (Abrams, Cruse, & Kunze, 2008). The new UC3 approach to digital curation infrastructure is based on the idea of devolving necessary function into a set of independent, but interoperable, micro-services that embody curation values and strategies. Since each of the services is small, they are collectively easier to develop, deploy, maintain, and enhance (Denning, Gunderson, & Hayes-Roth, 2008). Equally as important, since the level of investment in and commitment to any given service is small, they are more easily replaced when they have outlived their usefulness. Although the individual services are narrowly scoped, the complex function needed for effective curation emerges from the strategic combination of atomistic services (Fisher, 2006). Micro-services can be deployed in the contexts in which it makes most sense, both technically and administratively. While UC3 will use the micro-services as the basis for its ongoing centrally-managed curation activities, these services can also be The International Journal of Digital Curation Issue 1, Volume 5 | 2010 174 Stephen Abrams et al. usefully deployed and operated in local campus IT, research group, and departmental environments. It is no longer necessary that digital content must be transferred to a common repository in order to receive appropriate curation. Curation Micro-Services The UC3 curation micro-services are intended to achieve the following strategic goals reflective of evolving community best practice: • Providing safety through redundancy (embodying the principle that “lots of copies keeps stuff safe”; (Reich & Rosenthal, 2001)). • Maintaining meaning through description (“Lots of description keeps stuff meaningful”). • Facilitating utility through service (“Lots of services keeps stuff useful”). • Adding value through use (“Lots of uses keeps stuff valuable”). In consequence, the overall infrastructural framework is conceived in terms of an initial set of 12 micro-services arranged in four hierarchical service layers, each building upon the necessary foundational function of lower layers, and approaching curation sufficiency in the aggregate (see Table 1). Although the micro-services are assigned a mode and focus for purposes of classification, in actuality the services have broad applicability throughout the full curation lifecycle (see Figure 1). Mode Focus Layer / micro-service Curation Value Interoperation • Annotation • Notification Service Application • Transformation • Search • Index • Ingest Preservation Context Interpretation • Characterization • Inventory State Protection • Replication • Fixity • Storage • Identity Table 1. Curation micro-services. The Protection layer Identity and Storage services are foundational to the entire micro-services framework. The Identity service provides a means by which to persistently and unambiguously distinguish and reference a given unit of curated content. The Storage service provides a secure environment for the persistent management of that content. The Fixity service provides the means to detect damage to the bit-level integrity of managed content, and the Replication service manages the synchronization of content replicas. The International Journal of Digital Curation Issue 1, Volume 5 | 2010 An Emergent Micro-Services Approach to Digital Curation Infrastructure 175 Note that the four components of the Protection layer operate on content state without any understanding of what that content represents. The contextual meaning of curated content is managed by the higher-level Interpretation layer. The Inventory service maintains a comprehensive, schema-agnostic metadata catalog for the content managed in the Protection layer. The Characterization service provides an automated means to examine and extract the properties of formatted byte streams underlying managed content that are significant for purposes of curation and preservation analysis, planning, and intervention (Abrams, Morrissey, & Cramer, 2008). Discover / Use / Reuse Transform Create / Receive Appraise / Select Ingest Describe Store Monitor Transformation (derivative) Identity (resolve) Fixity Identity (bind) Identity (mint) Notification Annotation Search Storage Inventory Transformation (canonicalize) Characterization Inventory Index Annotation Curate Preserve